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ABSTRACT 



In November 2002, the Institute for the Advancement of 



Emerging Technologies in Education at AEL gathered education practitioners to 
help shepherd new assessments from the research laboratory to the classroom. 
Approximately 60 teachers and administrators attended the conference. The 
discussion focused on the possibility of aggregating formative data to meet 
the purposes of large-scale tests and attempted to develop a technology-based 
method of collecting and analyzing student outcome data to support testing 
decisions. A panel of forum presented offered an overview of research, 
specific examples of attempts to create new forms of assessment, and opinions 
about how the current state of research and the political climate combine to 
create an extraordinary opportunity to redesign assessments and make them 
more useful. Panelists agreed that the new systems require sophistication on 
the part of the teacher. Teachers and administrators in attendance wanted 
tools that were immediately useable, and it was evident that such tools do 
not yet exist. The gap between researcher needs and practitioner functioning 
is frustrating, but frustration is tempered by the attraction of the 
possibilities of new technology approaches. Panelists were: (1) John Bailey; 

(2) Linda Roberts; (3) Jim Pellegrino; (4) Eva L. Baker; (5) Christopher 
Dede; and (6) Lajeane Thomas. (SLD) 
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The State of Assessment 

Few computer users would purchase a database with no capability to organize and 
manipulate the various bits of data within it. Yet, we can see an analogous situation in the 
design and capabilities of our current student assessment systems. While current systems 
effectively gather data on student performance, they provide no means of interacting with 
the data or, in effect, making it meaningful. 

Research in cognitive science indicates that formative, diagnostic assessment that is 
embedded in instruction can substantially improve student outcomes. Such assessments, 
supported by advances in technology, can capture richer data than current systems — 
reflecting, for example, the process by which a student arrives at an answer instead of just 
the answer. In effect, new assessments can inform us of not only what students know but 
how they know it. 

New assessments can also measure students' depth of understanding by determining 
not just whether a student can recall a fact but whether the student can place that fact into 
a meaningful context. In addition, they can provide a timely definition of why a student’s 
understanding of a topic falls short of the goal. 

Such assessments require computational assistance to be manageable, however, and 
developing them is a complex task. Yet, addressing the complexities and working toward 
their development is of paramount importance if improved student outcomes are to be 
achieved. 

Moreover, assessments of the future must shift from the current emphasis on testing. 
Rather, they must be analytical tools, applied in a variety of media and formats, that capture 
multidimensional data on student achievement. These data can serve both formative and 
summative needs. While early research points to the potential value of such assessments, 
more funding is needed to continue this research. 

On November 12, 2002, The Institute for the Advancement of Emerging Technologies 
in Education (lAETE) at AEL gathered education practitioners to help shepherd these new 
assessments on the arduous journey from research lab to the classroom. The half-day 
forum, titled “Assessments That Empower Student Success: The Role of Technology,” was 
held in conjunction with the National School Boards Association’s Technology + Learning 
Conference, and drew from that gathering’s attendance of teachers and administrators. 
Approximately 60 teachers and administrators attended the LAETE forum. 
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Background 

The idea for this conference resulted from a January 2002 meeting of the lAETE 
advisory board. That discussion focused not on cognitive science research but on the 
possibility of aggregating formative data to meet the purposes of large-scale tests. lAETE 
wished to promote the development of a technology-based method of collecting and 
analyzing student outcome data that would simultaneously support decisions 

• by teachers about how to advance student learning 

• by district leaders about which schools and teachers were performing effectively 

• by states about which curriculum approaches were most effective 

• by the public and policymakers about the success of education 

The goal of using formative assessments for summative needs was originally put 
forward in Knowing What Students Know: The Science and Design of Educational Assess- 
ment, a report issued by the National Research Council (NRC). lAETE saw the potential to 
continue that work, the findings of which inextricably braid the assessment goal with 
cognitively based methods of teaching and assessment. A shift from creating tests to 
building analytical tools is the basis for the hope that data on student achievement can 
serve both summative and formative needs. 

This forum was intended to bring new understandings of how people learn and the 
great potential of technology-based systems for supporting and assessing learning to an 
audience of practitioners. Subsequent forums will bring these ideas to national meetings of 
researchers (AERA) and policymakers (to be announced). 

lAETE (www.iaete.org) promotes the purposeful use of new and emerging technolo- 
gies to improve teaching, learning, and school management. Since 1966, AEL (www.ael.org), 
a not-for-profit corporation, has integrated research, practice, and technology to spark 
innovation in education and promote strategies to increase student achievement. In 2000, 
it received a national leadership designation in educational technology from the U.S. 
Department of Education. 
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Forum Panelists 

The panel of forum presenters offered an overview of research, specific examples of 
attempts to create new forms of assessment, and opinions of how the current state of the 
research and the political environment combine to create an extraordinary opportunity to 
redesign assessments and make them more useful. 

John Bailey, director of the Office of Educational Technology at the U.S. Department 
of Education, kicked off the panel. Dr. Linda Roberts, who held the office in the previous 
administration, concluded the discussion. Both anticipate a new generation of technology- 
based assessments that offer richer insight into student knowledge. Dr. Roberts is now a 
consultant to companies, foundations, and governments. She also serves on the board of 
directors of several organizations, including Wireless Generation and the Sesame Work- 
shop. 

Dr. Jim Pellegrino, distinguished professor of cognitive psychology and education at 
the University of Illinois at Chicago, offered a synthesis of research. Dr. Pellegrino has 
chaired several National Academy of Sciences (NAS) and National Research Council (NRC) 
study committees, including the NRC committee that issued Knowing What Students Know. 
That book-length report is a companion to another resource referenced frequently 
throughout the lAETE forum. How People Learn. These reports, published by the National 
Academy Press, sparked the planning of this lAETE forum. 

Dr. Eva L. Baker presented her work on translating research findings into usable 
assessment systems. She is the current chair of NRC’s Board on Testing and Assessment. In 
addition, she is a professor of psychological studies and social research methods; 
codirector of the National Center for Research on Evaluation, Standards, and Student 
Testing (CRESST); and director of the UCLA Center for the Study of Evaluation. Dr. Baker, 
who also served on the NRC committee that created Knowing What Students Know, 
described technology-based assessments that teachers can easily create with computer 
support. She described assessments from CRESST’s knowledge map work that can be used 
with any subject matter domain. 

Dr. Christopher Dede also addressed applications of technology. Wirth Professor of 
Learning Technologies and chair of the Learning & Teaching Area at the Harvard Graduate 
School of Education, Dr. Dede described his research with virtual environments and the 
assessment implications of that work. Dr. Dede also was a member of the NRC committee 
that created Knowing What Students Know. 
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Dr. Lajeane Thomas, professor of educational computing and technology at Louisi- 
ana Tech University and chair of the International Society of Technology in Education 
(ISTE) Accreditation and Professional Standards Committee, discussed the assessment of 
teachers’ technology skills. With Dr. Roberts, she also led an audience discussion. 



The Political Context 

The one thing that No Child Left Behind did, for better or worse, was to really 
help elevate assessments in the debates and discussions about education and 
education reform. 

— John Bailey 



First of all, the truth of the matter is, assessment is hot. The public s attention is 
on assessment and accountability. It is an incredible opportunity for us to improve 
what we do. 

— Dr. Linda Roberts 



We are in a ‘reform* movement where powerful methods of teaching/learning 
are harder to use, due to flawed standards and tests. The only way to improve this 
situation is to give people something to move toward — not something to move 
against — because then we*ll Just react away from what we have now. into some 
other flawed method of reform. 

— Dr. Christopher Dede 



Bailey pointed out that critics of No Child Left Behind (NCLB) have redubbed it “No 
Child Left Untested.” He did not flinch when Dr. Dede referred to state assessments as 
“drive-by” tests. The NCLB federal mandates for state tests, Bailey made clear, do not imply 
that the administration is satisfied with current assessments. Said Bailey: 

Vm sensing, at least from both federal and state policymakers, a real hunger for 
new alternatives and new ways of assessing students. I have not met a single 
politician or public policymaker who is pleased with the full range of standardized 
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tests [available to them] today. They all recognize that these are very blunt tools. 
Unfortunately, what they believe, whether it’s perception or reality, is that those 
are the only tools that are available to them. 

Technology promises more sophisticated tools that, through automation, are also 
affordable on a wide scale. Even so, identifying how students progress through an 
understanding of a topic is not simple. It requires a great deal of work to define discipline- 
specific levels of understanding as well as general cognitive advances. Dr. Dede pointed to 
the substantial R&D investment that is essential to creating these assessments: “Front-end 
investments involve significant time and resources, both of which are scarce in this current 
context. However, we must find ways to bootstrap our research efforts if we’re going to 
develop an alternative model to current testing practices.” 

Assessments based on discipline-specific levels of understanding do have cost advan- 
tages, however. Researchers agree this work will establish a sustainable foundation that can 
be reused. Unlike current exams, which need a fresh set of test items for each round, 
insights to understanding endure and cannot be manipulated by students repetitiously or 
surreptitiously “finding” the right answer in advance. 



The Research Base 

What we know in terms of cognitive science says that assessment has to move 
beyond what we ve been doing for a long time — that is, assessing discrete bits and 
pieces of knowledge. We have to get at these more complex aspects of knowing and 
understanding — including metacognitive understanding, conceptual organiza- 
tion, and the ways in which students have represented knowledge that helps them 
solve problems. 

— Dr. Jim Pellegrino 



In every other field, research means continually pushing the envelope and 
continually testing the ideas with documentation, real observation, and a variety of 
experimental designs. Thats what we all have to keep asking for in education. . . . 
The public needs to commit to research in education the way we ha ve committed to 
research in other fields like cancer, like space technology. 

— Dr. Linda Roberts 
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Knowing What Students Know defines assessment as a process of reasoning from 
evidence. In Dr. Pellegrino’s words, “We can’t know what’s in a student’s head. We can 
only infer it from the kinds of situations that we present to them and the kinds of data that 
they give us in those situations.” The Knowing What Students Know report identifies an 
assessment triangle with three essential elements for inferring what is in a student’s head. 
That assessment triangle includes: 

• a model of student learning (cognition) in an academic domain 

• observation 

• an interpretation model 




Cognition 

Figure 1 . Assessment Triangle 



The components are interrelated and must be coordinated. The topic for this confer- 
ence was limited to assessment for the sake of a manageable discussion. In reality, new 
forms of assessment require new definitions of what it means to know and learn a subject. 

Current research uses new understandings of expert knowledge to define progress 
through and mastery of a topic. Again, as Dr. Pellegrino said: 

It turns out that experts just don’t know a lot of stuff. Yes, they know a lot of 
stuff, but what really defines expertise is how their knowledge is organized. It’s not 
that they know all these bits and pieces, but that the knowledge is organized into 
conceptual schemes that are extremely targeted for use. ... It is organized so that 
experts can fluently access and use this knowledge. They recognize patterns that, 
fundamentally, beginners don’t see. 
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Some implications for instruction are that teaching must be directed toward a gradual 
acquisition of understanding and expertise, and that facts and ideas must be understood by 
the student in the context of a conceptual framework. Meaningful comprehension and 
retention require scaffolding. Said Dr. Pellegrino, “We need to teach some subject matter in 
depth, providing lots of examples and providing firm foundation. But it doesn’t mean we 
have to teach everything.” 

There also appear to be learning patterns innate to our neural programming, and 
identifying these can guide cognitively based instruction and assessment. Work in early 
numbers, for example, shows a predictable path of understanding numbers. The ability to 
count on the nondominant hand, for example, is an ability distinct from counting on the 
dominant hand. Counting objects must precede a mental counting line. Indeed, there is 
likely a neurologically based number sense, much like phonological awareness, that leads 
to mathematical disabilities when not properly developed. 

A cognitively based model of student learning dovetails well with inquiry-based 
learning — “teaching in chords instead of notes,” as Dr. Dede says. Metacognition, or asking 
students to be increasingly aware of their own learning strategies, is also encouraged by 
cognitive scientists. Again, the researchers’ desire for exploration runs counter to current 
demands in the classroom to cover a broad range of precise content. Constructivism is not 
out of favor as a learning strategy; it is simply not a practical strategy for succeeding with 
current assessments. 

The key implication for assessment, said Dr. Pellegrino, is that it “has to be designed to 
capture these multiple components of organized knowledge — not just the bits and pieces 
but the conceptual schemes and how things hang together.” Furthermore, it is important to 
recognize that there are multiple paths toward competency. More important than defining 
the final state of knowledge is understanding a student’s specific strategy and where it falls 
on a continuum of efficiency and appropriateness for a given domain. This can be done 
through uncovering more complex aspects of knowing such as metacognition and 
conceptual organization. 
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To What End? Auditing School Performance or Defining Student Needs 



Dr. Pellegrino and Knowing What Students Know slv^ss that using summative data at 
the state level and formative data to guide classroom instruction require different tests. 
According to Dr. Pellegrino 

• assessment design should be based on a model of student learning 

• we should figure out ahead of time what inferences about student competence we 
want to make 

• inferences about student competence should be tailored to a particular context of use 

“It’s different if we’re doing something in the classroom for formative purposes than if 
we’re doing something at the state level for high-stakes assessment, or for summative or 
program evaluation,” Pellegrino said. “One of the mistakes we consistently make is that we 
think one kind of test can fit all three purposes. That is decidedly wrong.” 

However, given that Dr, Baker disagrees, research on that issue is not closed, 
“Essentially, I think the challenge in American education is to do what Jim says can’t be 
done — and that is to find tests that can be used for multiple purposes,” she said. Dr, Baker 
thinks a test can be designed from the beginning to provide data for both the classroom 
teacher and the policymaker, and that we can “find a way to credit what people do in class 
groups and in districts in the accountability system.” 

Dr. Baker added that her research team pays close attention to scalability and reusable 
attributes. She noted that there are certain attributes of cognition that transfer across subject 
matters and then there are certain things that have to be embedded in that subject matter 
domain. Obviously, the more domain independent the more reusable. Dr. Baker believes 
that more components of cognition are domain independent than does Dr. Pellegrino. 

She admitted there is more work to be done, saying, “Our interest has been trying to 
find the crosswalk — or ‘alignment’ — ^between and among standards and assessments, 
standards and instruction, and instruction and assessment. And the reality is, in my view, 
that nobody knows how to do it very well,” 

Dr, Roberts stressed that we need scientific research to move forward in the application 
of today’s technology and in the development of the next generation of tools and 
applications. She pointed to Scientific Research ton (National Academy Press) as 

a “must read” for academic researchers and school leaders because of its ability to help 
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readers understand (1) the range of scientific designs that must be deployed in different 
contexts to answer different questions; (2) the level of investment and number of studies 
over time that are required to build on both theory and effective practice; and (3) the need 
to report success and failure and to encourage debate, criticism, and reformulation of ideas 
and practice. 

“Research is a continual building of knowledge,” added Dr. Roberts, “not a point in 
time where you stop and you say, ‘Well, we know what we need to know about reading,’ 
or ‘We know what we need to know about teaching.’” 



Applied Research 

Now. it turns out that there are actually a number of examples that we can find 
in the literature of the use of cognitively based student models to design assessment 
practices that . . . can be woven into instruction. . . . There is work being done. 
However, it is only for pockets and parts of the curriculum. 

— Dr. Jim Pellegrino 

The new federal laws (e.g., NCLB) allow for diverse forms of assessment and multiple 
indicators of annual student progress. Bailey noted that the administration expected some 
states to move to portfolio assessment but that has not yet happened. He explained that this 
is partly due to problems with reliability, validity, and generalizability between and among 
schools and classes. “Again, I think, those are some areas that need some further thought 
and some further research development,” he said. 

Alternatives to multiple-choice or short-answer tests have long been available. Tech- 
nology has also offered alternatives in assessment independent of the insights to cognition 
that were the focus of this conference. Various kinds of embedded assessment are common 
in instructional software. Adaptive learning programs, also called individualized learning 
software, for drill-and-practice tutorials, were among the earliest software offerings for 
schools. Now, simulations serve as assessments for many professionals, including archi- 
tects, dental hygienists, and airline pilots. Software can also score writing automatically. 
Summary Street, from the University of Colorado (http://lsa.colorado.edu/summarystreet), 
is an example of software that analyzes how well a student has summarized a text from a 
collection in the program. The program is based on latent semantic analysis. Similarly, the 
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Educational Testing Service has developed software called E-rater to assess essays based on 
natural language processing (www.ets.org/research/erater.html). 

Now districts are preparing to merge such reports as they move to data-driven decision 
making. Dr. Baker worked on the Quality School Portfolio or QSP (http://qsp.cse.ucla.edu), 
software that facilitates aggregating and disaggregating an array of computer-generated 
data. One of her ongoing concerns, which is also an issue of this conference, is ensuring 
the value of the information stream used for decision making. 

Again, if the vision presented at this conference is to be realized, assessment must be 
redesigned with new definitions of what we want students to learn, new data to measure 
students’ progress, and a valid interpretation model. 

Speakers gave a glimpse of various strategies to turn findings from cognitive science 
into workable learning and assessment systems. The majority are pure research, but some 
are already usable products. These strategies tend to include the added benefit of 
simultaneously helping teachers to learn more sophisticated assessment practices. 

• Dr. Baker and her team at CRESST are working to create an authoring system for 
classroom assessment. It uses computer-based concept mapping to define how 
students connect ideas, and the software can then compare student maps to those 
made by people at various levels of expertise (www.cresst.org). 

• Through a NSF-funded research project. Dr. Dede and his Harvard team are creating 
and evaluating a graphical multi-user virtual environment experiential simulator 
(MUVEES) to teach middle school students experimental design. This simulator 
provides a gamelike context for students to explore water quality problems in a 
nineteenth-century town called River City (www.gse.harvard.edu/'-dedech/muvees). 

Dr. Dede stated, “Developers must begin by creating a really powerful learning 
experience, then sweating blood figuring out how to assess it.” The model of learning his 
group is exploring is based on guided inquiry, modeling and visualization, and virtual 
collaboration. For assessment, the program can generate a great deal of data about student 
decisions — perhaps too much. To be able to make sense of the data flow, said Dr. Dede, 
“We need the insights of practitioners to understand the interpretive issues that are 
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involved. Working in partnership with an expert teacher or teachers can help us disen- 
tangle what’s happening in the minds of the students.” 

• Wireless Generation, a New York company (www.wgen.net), has created a handheld 
device that reading teachers use to make notations of a student’s progress. Synched 
on a computer, the system analyzes the data. The product meets the often-stated goal 
of letting technology deliver data so that teachers can focus on interaction with 
students. 

• Several speakers referenced Jim Minstrell’s facets-based instruction for physics, 
developed after he had taught high school physics for 30 years. Minstrell, said 
Pellegrino, defines a facet as “a convenient unit of thought, an understanding or 
reasoning, a piece of content knowledge or strategy seemingly used by the student in 
making sense of a particular situation.” A preinstruction quiz identifies facets. That is 
followed by instruction with benchmark lessons to address a student’s specific 
misconceptions and preconceptions. Students also practice problems on a computer 
program called the Diagnoser, which holds carefully designed problems to probe 
how well the student is understanding (http://depts.washington.edu/huntlab/ 
diagnoser). 

• Interactive Multimedia Exercises, or IMMEX, gives students carefully designed, 
complex problems and captures how they are making choices. The program then 
compares students’ paths of moving through the problem space to good solutions, 
intermediate solutions, and weak solutions using artificial neural network analysis. 
This is the work of Ron Stevens of UCLA (www.immex.ucla.edu). 

• Cognitive Tutor, a collection of mathematics software products, makes use of 20 years 
of research in cognitive science led by Dr. John R. Anderson at Carnegie Mellon 
University. The software analyzes students’ mistakes and presents them with appro- 
priate problems, hints, and tutorials (www.carnegielearning.com). 
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New Assessment Technologies Require Teacher Sophistication 



These systems require a tremendous amount of sophistication by the teacher. 
Inquiry learning in general requires deep instructional skills because teachers have 
to understand content deeply and the learning process is more complex than with 
presentational/assimilative teaching. When this dimension of interpretive assess^ 
ment is added, teachers really need to understand students in order to be able to see 
inside of their heads and comprehend what they re doing. The good news is that 
systems like this are an excellent way of getting teachers to think about their 
teaching as well as learners to think about their learning. 

— Dr. Christopher Dede 



The experts participating in this panel all anticipate a day when teachers, in Dr. 
Roberts’ words, “have the right information at their disposal, at the right time, so that they 
can be the coaches, the mentors, the role models for their students.” 

Reaching this point requires that researchers and expert teachers work together to 
define which information about student learning is important. It is, for example, now 
possible to track students’ eye movements on a computer screen as well as their keyboard 
and mouse input. In what circumstances is such data of value? Furthermore, teachers need 
to have relevant information rapidly pulled from the torrential stream that can be 
generated. 

Yet, however technologically sophisticated, a refined system will continue to ask a 
great deal of teachers. Dr. Dede explains that people once thought technology would allow 
them to create a personal tutor in a box. That concept, he said, is “beyond the capabilities 
of artificial intelligence now and probably at any point in the reasonable future.” But, adds 
Dr. Dede, “we can aggregate information in a way that makes it easier for a teacher to judge 
the current stage of a student’s understanding. Through this, we can gain leverage on the 
enormously complex challenge of improving learning and assessment.” 
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Practitioner Response to the Conference 



When participants were asked via questionnaires what related topics they would 
suggest AEL investigate, their responses included “In all of this I see nothing about 
sustainability, replicability, teacher training, nuts & bolts issues” and “specifics that work.” 

Panelists noted that the event clarified the distinct interests of the groups that had 
gathered. Policymakers need accountability, teachers want improvement to the current 
curriculum, and researchers seek transformation. 

Clearly, teachers, principals, and other practitioners in attendance wanted something 
immediately usable. A key question about the MUVEES, for example, was whether it was 
designed to cover science, writing, and social studies standards. Given the immense 
pressures on classroom time to cover the standards, could students’ use of this tool ensure 
that the items were still covered? 

The answer is not yet. This is a research project, and close integration with standards 
is beyond its current scope. At this stage. Dr. Dede is not building a product to bring to 
market; he is testing ideas about learning and teaching. 

Indeed, the gap between researcher needs and practitioner needs is frustrating to those 
on the front lines. While each speaker emphasized the need to develop tools in tandem 
with teachers, and to seize the moment for a hue and cry to fund further research, the 
practitioners in the audience wanted an answer for this semester. They also asserted that 
their classroom demands may not permit time for reflection to assist researchers, or for the 
activism required to fund them. 

Rooted in reality, the practitioners in attendance also knew they must be able to answer 
to the psychometricians. Dr. Baker pointed out that researchers focused on cognition and 
learning are in a decidedly different camp than the psychometricians focused, for the most 
part, on the technical quality of tests. Said Baker, “Technical quality is necessary but 
nowhere near sufficient if instructional improvement is desired.” 

Teachers were also wary of yet another software program that required more training. 
Commented one attendee, “It seems to me that we’ve gotten to the point that it’s absolutely 
impossible to train any group of people in all . . . that we have to train them in. It has just 
reached a sort of point of absurdity.” Dr. Roberts agreed, saying, “I think training is the 
wrong term here anyway. I think that what we really want to do with the teachers is build 
their expertise and give them competent tools and opportunities to use that expertise.” 
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Dr. Pellegrino added that schools should require publishers to embed some of these 
tools in curriculum materials “to put in [teachers’] hands good tools and let them practice 
what they know well, which is how to help kids learn.” 

Of course, curriculum publishers will only follow the market. As Dr. Baker said, “If we 
continue to do what we’ve done — have classroom and school formative evaluation strand 
go this way and external mandated tests go that way — there’s going to be zip incentive, 
except idealism for people that hang in there with the more interesting assessments. . . . 
People are pushing towards this focus on external accountability.” 

Overall, the frustration exhibited by teachers and administrators was tempered by a 
strong attraction to the possibilities. There were many enthusiastic responses, and one 
attendee put it this way, “Please continue to keep this topic in the public eye.” 



Resources for Learning More 
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